Add CI check that the previous release works against the new schema#3506
Conversation
|
Claude finished @jonathangreen's task in 5m 23s —— View job Code Review
SummaryThis is a well-constructed, self-contained addition that faithfully mirrors the established
The earlier review threads (checkout pinned to No changes requested from me. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3506 +/- ##
==========================================
+ Coverage 93.45% 93.46% +0.01%
==========================================
Files 511 512 +1
Lines 46533 46614 +81
Branches 6350 6352 +2
==========================================
+ Hits 43488 43570 +82
+ Misses 1969 1968 -1
Partials 1076 1076 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…3507) ## Description Makes four tests robust to running serially with pytest-xdist disabled (`-n0`). They pass under the normal xdist test run but fail under `-n0`, which is how the backwards-compatibility CI check (#3506) runs a previous release's database suite against the current schema. - **Three script tests** (`test_equivalents`, `test_novelist`, `test_search`) constructed a `Script` with no `cmd_args` and called `do_run()`, so the script's argparse fell back to `sys.argv[1:]`. Under xdist each worker has a clean execnet `sys.argv`, but under `-n0` `sys.argv` is the real pytest command line. That caused argparse to error on unrecognized arguments (`equivalents`, `novelist`), and the leaked `-m db` marker matched `RebuildSearchIndexScript`'s `-m/--migration` flag, sending it down the migration path so `search_reindex.delay` was never called (`search`). Each now passes an explicit `cmd_args=[]` to express "no arguments" independent of `sys.argv`. - **The OIDC logout test** (`test_oidc_logout_initiate_no_stored_id_token`) asserted on `caplog` without setting a capture level. Under serial execution the effective log level is left raised by an earlier test, so the warning was not captured and `caplog.text` was empty. It now sets the level explicitly, matching its passing sibling `test_oidc_logout_initiate_revocation_only`. No production code changes — scripts intentionally default `cmd_args` to `sys.argv` for real CLI use via `Script.run()`. ## Motivation and Context The backwards-compatibility check added in #3506 runs the previous release's `-m db` suite under `-n0` (external-schema mode requires serial execution — the single shared database can't be used by parallel xdist workers). Its first real run surfaced these four latent test-isolation bugs, which xdist had been masking. None are schema-compatibility failures; all four fail identically against the previous release's own schema. Fixing them here means that once a release containing these fixes becomes the "previous release", the check will pass against it. ## How Has This Been Tested? Reproduced the failures on the unfixed tests by running them under `-n0` in the docker test environment (`tox -e py312-docker -- -n0 ...`), observing the argparse `unrecognized arguments` errors. With the fixes applied, all four pass under `-n0` (including with `-m db` present, which exercises the `search` flag-collision case): `60 passed, 19 deselected`. ## Checklist - [x] I have updated the documentation accordingly. - [x] All new and existing tests passed.
|
This is going to fail until #3507 gets into a release, so I'll leave it in draft for now. |
Add a backwards-compatibility CI job that runs the previous release's database tests (pytest -m db) against the schema produced by the current code, exercising the online-migration requirement that a release's database must keep working with version N-1 of the code while its webservers are still serving traffic. - docker/ci/test_backwards_compatibility.yml: compose overlay defining a webapp-prev service (the previous release's image) in external-schema mode. - docker/ci/test_backwards_compatibility.sh: resolves the previous release from the latest GitHub release, builds the new schema with the current image, then runs the previous release's db tests against it. Skips when the previous release predates the external-schema seam. - build.yml: new backwards-compatibility-test job, added to the push job's needs. This is the second of two PRs and depends on the test seam from the first. Until a release containing the seam becomes the previous release, the check skips.
…bility job The previous release now ships the external-schema test seam, so the job no longer needs to detect older images that lack it and skip.
18b6dbb to
4512a47
Compare
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Flip the previous-release test run from serial (`-n0`) to parallel (`-n auto`) now that external-schema mode can clone the externally-built schema into a per-worker database. The compose overlay enables database creation (`PALACE_TEST_DATABASE_CREATE_DATABASE=true`) so each xdist worker gets its own clone of the schema built by the current image. This depends on the parallel-capable external-schema fixture, which ships separately. Because the job runs the *previous release's* baked-in test code, this change must not merge until a release containing that fixture is the previous release -- so keep this PR in draft until then.
The backwards-compatibility check resolves the previous-release image from the latest GitHub release. On a release build the just-published tag is the latest release, so the script resolves to this build's own X.Y.Z image, which the push job has not created yet — the pull fails and the release deadlocks. Skip both gated tests on tag builds via a step-level guard so the jobs stay green and the push job's dependency is satisfied. The tagged commit was already exercised by these checks on its branch/main push, so no coverage is lost.
…resolved The previous-release lookup ran gh with 2>/dev/null and inferred "skip" from empty output, so any failure -- a genuine absence of releases or a transient auth/rate-limit/network error -- was treated identically and let the gate no-op. The backwards-compatibility check would then fail open silently, and push would proceed without it ever running. Let gh's error flow to the job log, check its exit status explicitly, and emit a GitHub Actions ::warning:: on any failure before skipping. We still don't fail the job (a blip here shouldn't block releases), but the skipped gate is now visible at the PR/checks level instead of passing quietly.
| # push, so re-running here is redundant. See backwards-compatibility-test for why | ||
| # release builds must not run these gated tests. | ||
| - name: Test migrations | ||
| if: ${{ !startsWith(github.ref, 'refs/tags/') }} |
There was a problem hiding this comment.
This skip isn't strictly necessary, but I added it for consistency with the backwards-compatibility-test skip below. Since we shouldn't need these tests for a tagged release build.
The README attributed the backwards-compatibility CI check to the PALACE_TEST_DATABASE_CREATE_DATABASE=false / serial path, but the check runs with database creation enabled and pytest -n auto (parallel), matching the external-schema-with-creation scenario described just above. Move the attribution to that scenario and leave the create-database=false / serial option as a general alternative.
## Description Fixes the `backwards-compatibility-test` job, which has been failing on `main` since #3506 merged, with: ``` no such service: redis ##[error]Could not start service containers. ``` `docker/ci/test_backwards_compatibility.sh` asked docker compose to start a `redis` service. #3498 renamed that compose service to `valkey`. This renames the reference. The service is still needed, so this is a rename rather than a removal: `initialize_instance` itself only takes a Postgres advisory lock, but the `webapp-prev` container inherits `PALACE_TEST_REDIS_URL: redis://valkey:6379/3` from the `webapp` service, so the container has to be up for the previous release's test run. The `redis://` URL scheme and the `PALACE_*REDIS*` variable names are deliberately left alone, consistent with #3498 — they name the protocol, not the instance. ## Motivation and Context #3506 was branched off `main` before #3498 landed, and was never rebased. At #3506's head commit `docker-compose.yml` still defined a `redis` service, so the script was correct on the branch and the job passed there. The two PRs touch disjoint files, so git merged them without a textual conflict — the conflict was semantic, and nothing surfaced it until the job ran on `main`. This also means the `push` job, which now depends on `backwards-compatibility-test`, is blocked on `main` until this lands. ## How Has This Been Tested? Reproduced the failure and confirmed the fix locally with `docker compose ... up --dry-run`, which validates service names without starting containers: - with the old `redis` argument: exit 1, `no such service: redis` — the identical message from the failing CI log. - with the new `valkey` argument: exit 0. Then started the service for real and confirmed it reaches `healthy` and that `valkey-cli ping` answers `PONG`. I did not run the script end-to-end locally, since that requires pulling both the current and previous release images and running the full `pytest -m db` suite. Worth flagging for review: this fix unblocks a step that has never actually executed, because the previous release's test suite will now run against Valkey for the first time — on #3506's branch it only ever ran against `redis/redis-stack-server`. I expect that to be fine (Valkey is protocol-compatible, `main`'s own db suite already passes against `valkey:9.0` in the tox stack, and the previous release's Redis usage is core commands only), but this job is where that combination gets exercised first. The job runs on non-tag refs, so this PR's own CI is the real verification. ## Checklist - [x] I have updated the documentation accordingly. - [x] All new and existing tests passed.
Description
Adds a CI job,
backwards-compatibility-test, that verifies the previous released version of the application still works against the database schema produced by the current code. This enforces our online-migration requirement: a release's database must keep working with version N-1 of the code, because new migrations run while the previous version's webservers are still serving traffic during a deploy.The check works by:
circ-webappimage.pytest -m db) against that schema in external-schema mode (PALACE_TEST_DATABASE_EXTERNAL_SCHEMA), so the older code exercises the new schema. If those tests fail, the migration is not backwards compatible.New files:
docker/ci/test_backwards_compatibility.sh— orchestrates the check.docker/ci/test_backwards_compatibility.yml— compose overlay adding awebapp-prevservice that runs the previous release's image in external-schema mode.The
pushjob now depends onbackwards-compatibility-testso images are not tagged and pushed unless the check passes.Motivation and Context
Our migrations are run online, while the previous version of the application is still serving traffic, so a release's database must work with version N-1 of the code. Until now nothing in CI verified that invariant — a migration that broke backwards compatibility would only be caught during a deploy. This job catches such regressions before images are pushed.
How Has This Been Tested?
The check is itself a CI job and runs as part of this PR's build against the latest published release image. The script also supports a
PREV_RELEASE_IMAGEoverride for running it locally against a chosen previous image.Checklist